This invention relates generally to testing methodologies and more particularly to metrics-related testing of an Operational Support System (OSS) of an incumbent provider for compliance with a regulatory scheme.
In the United States, a regulatory entity of a state or other jurisdiction may be tasked with considering whether, and to what extent, an ILEC is in compliance with the requirements of Section 271 of The Telecommunications Act of 1996 (xe2x80x9cthe 1996 Actxe2x80x9d). The 1996 Act, together with associated Federal Communications Commission (FCC) interpretations, generally require an ILEC seeking to enter the market for long distance telecommunications services to: (1) provide the relevant CLEC community with non-discriminatory access to the ILEC""s OSSs, subject to suitable terms and conditions; (2) provide documentation and support necessary for CLECs to use these OSSs; and (3) demonstrate that these OSSs are operationally ready and provide an acceptable level of performance to the CLEC community. Compliance with these requirements should, for example, allow CLECs to obtain pre-ordering information, fulfill orders for resale services and unbundled network elements (UNEs), manage problems, and obtain billing information in a manner that is non-discriminatory as compared to the ILEC""s own retail operations.
The regulatory entity may retain an independent testing entity to test and evaluate the readiness of the ILEC""s OSS interfaces, documentation, and processes to support local market entry by CLECs. Based on its testing activities, the testing entity may generate a report documenting the manner in which its testing activities were conducted, results of the testing activities, and its evaluations based on the results. The audience for the report will generally fall into two main categories: (1) readers who will use the report in connection with the regulatory process, and (2) any other interested parties who may have some stake in the matter and wish to have visibility into the testing activities, results, and evaluations (e.g., the ILEC being evaluated, the CLECs, and other ILECs). While many of the above parties may have some interest in the outcome of the testing activities, typically only the regulatory entity and the ILEC are parties to the contract with the testing entity.
According to the present invention, problems and disadvantages associated with testing OSSs of an incumbent provider may be reduced or eliminated.
In one embodiment of the present invention, a method for testing one or more OSSs of an incumbent provider for compliance with a regulatory scheme is performed by an independent testing entity attempting to emulate a competitive provider that would access the OSSs in attempting to compete with the incumbent provider in an open competitive market. The method includes conducting active testing of the OSSs with respect to one or more variables and evaluating performance of the incumbent provider during active testing with respect to the one or more variables according to predetermined evaluation criteria comprising one or more performance metrics. Evaluation of incumbent provider performance includes, for performance metrics requiring the incumbent provider to meet a particular performance standard, using a one-sample t-test for continuous variable performance metrics and a binomial test for discrete variable performance metrics. For performance metrics requiring parity between incumbent provider performance for operations of the incumbent provider associated with its interaction with competitive providers and incumbent provider performance for its internal operations, a two-sample t-test is used for continuous variable performance metrics and a hyper-geometric test is used for discrete variable performance metrics. The method further includes generating test results according to the evaluation and issuing a final report comprising the test results and providing a sufficient basis for a regulatory entity administering the regulatory scheme to determine compliance of the incumbent provider with the regulatory scheme.
In another embodiment of the present invention, a method for testing one or more OSSs of an incumbent provider for compliance with a regulatory scheme is performed by an independent testing entity attempting to emulate a competitive provider that would access the OSSs in attempting to compete with the incumbent provider in an open competitive market. The testing is associated with metrics capture, tracking, and reporting activities of the incumbent provider with respect to a particular operational domain. For testing of an OSS, the method includes within a data integrity testing component during active testing, to determine whether filtered data the incumbent provider uses to calculate values for each of one or more metrics is accurately derived from raw data the incumbent provider collects in connection with transactions associated with the OSS: (a) identifying one or more fields in the filtered data that the incumbent provider uses in calculating values for the metric; (b) identifying a random sample of transactions associated with the OSS; (c) obtaining from the incumbent provider copies of records associated with the identified transactions, the records containing raw data for these transactions from which filtered data for these transactions in the identified fields was derived; and (d) for each identified field, determining whether the filtered data for the identified transactions in the identified field is consistent with the raw data for the identified transactions contained in the associated records. The method further includes within a metrics replication testing component during active testing, for each of one or more metrics for which the incumbent provider has calculated values: (a) obtaining from the incumbent provider a description of the metric, filtered data on which the incumbent provider based its calculation of values for the metric, and an algorithm according to which the incumbent provider made its calculation of values for the metric, the filtered data being associated with a specified time period, the filtered data being derived from raw data the incumbent provider collected in connection with transactions associated with the OSS; (b) re-calculating values for the metric according to the description, data, and algorithm obtained from the incumbent provider; and (c) determining whether the re-calculated values are consistent with the values calculated by the incumbent provider. The method further includes evaluating performance of the incumbent provider during active testing according to predetermined evaluation criteria, generating test results according to the evaluation, and issuing a final report comprising the test results and providing a sufficient basis for a regulatory entity administering the regulatory scheme to determine compliance of the incumbent provider with the regulatory scheme.
In another embodiment of the present invention, a method for testing one or more OSSs of an incumbent provider for compliance with a regulatory scheme is performed by an independent testing entity attempting to emulate a competitive provider that would access the OSSs in attempting to compete with the incumbent provider in an open competitive market. The testing is associated with performance metrics capture, tracking, and reporting activities of the incumbent provider across a plurality of operational domains. The method includes conducting a metrics data collection and storage test during active testing to evaluate incumbent provider policies and practices for collecting and storing data used for calculating performance metric values, both for internal operations of the incumbent provider and for operations of the incumbent provider associated with its interaction with competitive providers. The method also includes conducting a metrics definition documentation and implementation test during active testing to evaluate incumbent provider policies and practices for documenting and implementing performance metric definitions, implementation of performance metrics definitions comprising applying exclusions and business rules in calculating performance metric values based on raw data. The method also includes conducting a metrics change management test during active testing to evaluate incumbent provider policies and practices for developing, implementing, monitoring, and reporting changes concerning performance metrics, conducting a metrics data integrity test during active testing to evaluate incumbent provider policies and practices for processing data necessary to calculate performance metric values, and conducting a metrics calculation and reporting test during active testing to evaluate policies and practices of the incumbent provider for calculating performance metric values. The metrics calculation and reporting test includes (a) accessing calculated performance metric values for operations of the incumbent provider associated with its interaction with competitive providers, these values having been calculated by the incumbent provider and thereafter reported to the regulatory entity; (b) re-calculating the accessed performance metric values; (c) comparing the re-calculated values with the accessed values to identify any discrepancies; (d) attempting to reconcile any identified discrepancies between the re-calculated values and the accessed values; and (e) determining whether the incumbent provider calculated the accessed values consistent with documented instructions for doing so. The method also includes conducting a statistical evaluation of transactions test during active testing to evaluate metrics-related service performance of the incumbent provider according to statistical techniques. This test includes accessing test performance metric values calculated based on interaction between the incumbent provider and the testing entity, accessing benchmark performance metric values calculated based on interaction between the incumbent provider and one or more competitive providers, and comparing the test performance metric values with the benchmark performance metric values according to statistical techniques to determine whether and to what extent parity exists between the compared values. The method further includes evaluating performance of the incumbent provider during active testing according to predetermined evaluation criteria, generating test results according to the evaluation, and issuing a final report comprising the test results and providing a sufficient basis for a regulatory entity administrating the regulatory scheme to determine a net impact of identified metrics-related issues on the ability of competitive providers to compete with the incumbent provider in an open competitive market.